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Abstract 

It has been recently proved that the redundancy r of any discrete memo- 
ryless source satisfies r < 1 — fi(p N ), where p N is the least likely source letter 
probability. This bound is achieved only by sources consisting of two letters. 
We prove a sharper bound if the number of source letters is greater than two. 
Also provided is a new upper bound on r, as function of the two least likely 
source letter probabilities which improve on previous results. 



*MIT Laboratory for Computer Science, 545 NE Technology Square, Cambridge MA 02139, USA. 
This work has been done while the author was at the Dipartimento di Informatica ed Applicazioni, 
Universita di Salerno, 84081 Baronissi (SA), Italy. E-mail: robdepfitheory.lcs.mit.edu 

^Dipartimento di Informatica ed Applicazioni, Universita di Salerno, 84081 Baronissi (SA), Italy. 
E-mail: adsfidia.unisa.it 



1 Introduction 



Let S = {ai, a 2 , ..., ajv} be a discrete source with N letters and let pk denote the 
probability of letter a^, 1 < k < N. We assume, without loss of generality, that 
Pi > P2 > ••• > P N - Let C = {xi, x 2 , . . . , xjv} be a code for source S and let 
n^ < ri2 < . . . < n N he the codeword lengths. Codeword X{ encodes the letter a,-, 
for z = 1, 2, . . . , N. The length vector of a code C is the vector (rai, n 2 , . . . , n, N ). The 
Huffman encoding algorithm [6] provides an optimal prefix code C for the source S. 
The encoding is optimal in the sense that codeword lengths minimize the redundancy 
r, defined as the difference between the average codeword length L and the entropy 
H(pi 7 p 2} . . . ,p N ) of the source: 



N 



N 



L -H(p 1 ,p 2 , 



>Pn 



J^Pi^i + J2Pi^°gPi 



8 = 1 



8 = 1 



where log denotes the logarithm to base 2. It is well known that the redundancy of an 
optimal code satisfies < r < 1. These bounds can be improved if one knows partial 
information on the source. Gallager [4], Johnsen [7], Capocelli et al. [3], Capocelli 
and De Santis [I], and Manstetten [10] considered the problem of upper bounding r 
when pi is known. 

Reza [9] and Horibe [5] considered the problem of upper bounding the redundancy 
when the least likely source letter probability p N is known. Their bound has been 
recently improved by Capocelli and De Santis [2] who proved that, as function of p N , 
the redundancy r of Huffman codes is upper bounded by 



r < 1 - H(P N 



:v. 



where Ti is the binary entropy function 7i(p) = —plogp — (I — p) log (I — p). 

Recently, Yeung [13] introduced the notion of local redundancy and, by exploit- 
ing it, he derived the following bound as function of two least likely source letter 
probabilities: 



r < I -H{p N _ 1 ) + {p N _ 1 +p N ) 



i-n 



Pn 



A_! + P N 



Bound (I) is clearly the best possible upper bound on r as function of only p N . 
Indeed, any binary source (I — p N7 p N ) satisfies (I) with equality. If the source is not 
binary, the bound (I) is no more the best possible. 

In this paper we derive upper bounds on r as function of p N when additional 
information on the total number of source letters is known. 



We prove that for N > 3 the following bound, as function of the least likely source 
letter probability, holds: 

(l-H(2p N ) } H0<P N <S 

-\0.5 + 1.5p N -H(p N ), HS< Pn <1/3 {6) 

where 6 ~ 0.1525. This bound is the best possible expressed only as function of p N 
for every p N > and N > 3. 

We also present three tight upper bounds as function of the least likely source 
letter probability, when it is known that there are exactly three, four and five source 
letters. 

Finally, we use (3) to obtain an upper bound as function of the two least likely 
source letter probabilities which improves Yeung's bound (2). 

This paper is organized as follows. In Section 2 we present some definitions and 
results. In Section 3 we provide tight upper bounds as function of p N for the special 
cases N = 3,4,5. By using these bounds, in Section 4 we prove (3). Finally, in 
Section 5 we prove the upper bound on r, as function of p N _ x and p N . 



2 Preliminaries 

In this section we present some definitions and results, that will be useful in the rest 
of the paper. 

An Huffman code can be represented as a rooted tree in which each source letter 
corresponds to a leaf on the tree and where the associated codeword is the sequence 
of labels on the path from root to leaf. The code tree is generated by the Huffman 
algorithm. Each internal node in the code tree has a probability defined as the sum of 
the probabilities of his two children. The external nodes have the probability of the 
corresponding source letter. Gallager [4] proved that an Huffman code tree has the 
sibling property, i.e. each node, except the root, has a sibling and all nodes can be 
listed in order of decreasing probability with each node being adjacent to its sibling. 
We number all the nodes, except the root, using the sibling property, so that for each 
k } k = 1, . . . ,N — 1, nodes 2k — 1 and 2k are siblings, and the probability of a node 
i is greater than the probability of node i + 1, for each i = 1, . . . , 2 N — 3. Let qi be 
the probability of node z, i = 1, . . . , 2N — 2. The list (gi, . . . , g 2 JV-3, 92JV-2) is called 
the sibling list [4]. 

Let u be a node at level / in the Huffman code tree, and v a node at level / + 1. 
Then 

q u > q v - (4) 



Lemma 1 The redundancy r of a source, whose most and least likely source letter 
probabilities are respectively pi and p N , is upper bounded by: 

r < Pl + 0.086 - p N for < Pl < 1/6 (5) 

r < 2 - 1.3219(1 -pi)- H{jpi) - p N for 1/6 < Pl < 0.1971 (6) 

r < 4 - 18.6096pi - ft(5pi) - p N for 0.1971 < Pl < 0.2 (7) 

r < 2 - 1.25(1 - Pl ) - H{px) - p N for 0.2 < Pl < 0.3138 (8) 

r < 3 - (3 + 3 log 3)p x - H{3 Pl ) - p N for 0.3138 < Pl < 1/3 (9) 

r < 1 + 0.5(1 - Pl ) - Hipr) - 2p N for 1/3 < Pl < 0.4505, and N > 6 (10) 

Proof. Bound (5) is provided by Capocelli et al. [3]. 

Bounds (6) and (7) follow from Manstetten's work [10]. Manstetten proved that for 

Pi G [l/(j + 1), l/j] the redundancy can be written as 

m JV-1 

r = S + Y, l'k ~ H Wli ■ ■ ■ > ?m) + Y, (?2fc-l + ?2fc)[l - 'H{q2k/{q2k-l + ?2fc))], 

A;=t; k=m 

where s is the integer such that 2 s — 1 < j < 2 S+1 — 1, and q' l7 . . . } q' m are the entries 
<7j, . . . , ^2j, in the sibling list so that m = j ' + 1, and 5' = qi+j-i for z = 1, . . . , ra, and, 
finally, u = 2 S+1 — j. The above expression of r can be upper bounded by 



r 






Manstetten provided upper bounds on 5 + Y^k= v <?fc — H(q[ } . . . , q' m ) -\- q' m , as function 
of pi. From these bounds we can obtain bounds on r as function of pi and p N . In 
particular for j = 5,6, one obtains (6), (7) and (8). 

Bound (9) are due to Capocelli and De Santis [1]. Bound (10) has been obtained by 
Capocelli et al. (see Theorem 5 and subsequent remark in [3]). Q 

A useful result due to Montgomery and Kumar [12] is the following: 

Lemma 2 Let S be a discrete memoryless source whose most likely source letter 
probability is p^. If for some integer m 

2 1 

<Pi < 



2m + l _|_ \ " - rl " 2 m — 1 ' 

then an optimal code for S must have the minimum codeword length rt\ = m. Fur- 
thermore, if 

1 2 

<Pi < 



Om+l _ 1 — - rl " 9 m + 1 -I- 1 ' 

then an optimal code for S must have the minimum codeword length either rt\ = m 
or Hi = m -\- 1. 



3 Upper bounds on r as function of p , for fixed N 

In this section we provide three upper bounds on r, as function of p N when we know 
that the source has exactly 3,4,5 letters. More precisely, in subsections 2.1, 2.2, 2.3 
we consider the cases N = 3,4,5, respectively. 

These three upper bounds will be useful to determine the general upper bound 
(3) on r as function of p N , when N > 3. 

3.1 An upper bound when N = 3 

In this section we derive the upper bound as function of the least likely source letter 
probability, for ternary sources. 

Theorem 1 Let S = (pi 7 p 2} p 3 ) be a discrete source and p 3 = p N be its least likely 
source letter probability. The redundancy of the corresponding Huffman code is upper 
bounded by: 

(l-H(2p N ), tfO<p N <6 

-\0.5 + 1.5p N -H(p N ), ifS<p N <l/3 [ > 

where 6 ~ 0.1525 is the unique zero of the function 0.5 — 1.5a; — 7i(2x) + 7i(x) in the 
interval [0,0.3]. The bound is tight. 

Proof. The Huffman code for S has length vector (1,2,2), and its redundancy is 

r = 1 + p 2 + p 3 ~ H(p 1 ,p 2 ,p 3 ). 

Maximizing over all possible values for pi and p 2} given that p 3 = p N7 it follows 

r<l+p N + t max {y 2 -H(y 1 ,y 2 ,p N )} 

(yi,y2,p N )sQ 

where Q is the set of all source of three letters whose least likely source letter proba- 
bility is p 3 = p N , i.e. Q = {(yi,y 2 ,p N )\yi > Vi > p N ,yi + Vi + P N = 1}- Alternatively, 
the set Q can be written as Q = {(1 — x — p N7 x } p N )\p N < x < (1 — p N )/2}. Since 
the function x — Hil — x — p N7 x } p N ) is a convex U function of x the maximum value 
must occur at an extreme point of x's interval of variation. The extreme points of this 
interval are p N and (1 —p N )/2. If x = p N then r < 1 — 7i(2p N ) whereas if x = (1 —p N )/2 
then r < 0.5 + 1.5^ — 7i(p N ). Hence, we have 

r < max{l - H(2p N ), 0.5 + l.5p N - H(p N )} 

that leads us to (11). 

The bound is reached by the source (1 — 2p N7 p N7 p N ) } if < p N < <5, and by the 
source (I^l, 1 -^,p N ), if S<p N < 1/3. D 



Define the function ct(p N ) as follows 

n ( n ) = I l ~ U{2X) if ° < P - ~ 8 

[Pn) \0.5 + 1.5-H(x) if S< Pn < 1/3 

that is, the function a represent the bound on r given by (11). 

The function 1 — 7i(2x) is a decreasing function of x, for x £ [0, S]. The function 
0.5 + 1.5a; — 7i(x) is a convex U function of x with a minimum at x - 0.26, the 
unique zero of the first derivate of 0.5 + 1.5 — 7i(x) in [<5, 1/3]. Thus, the bound (11) 
is a decreasing function of p N for p N < \ an d it is an increasing function of p N for 
X < Pn — 1/3- Hence the following lemma holds. 

Lemma 3 The function ct(p N ) is a decreasing function ofp N , for p N £ [0,x] ; X — 
0.26, whereas for p N £ [x, 1/3], ct(p N ) is an increasing function of p N . 

Observe that, since a(6) > a(l/3), Lemma 3 implies that if p £]0,<5] one has 
ct(p) > a(x) for all x £]p, 1/3]. 

Bound (11) is depicted in Figure 1. 

3.2 An upper bound when N=4 

In this section we derive an upper bound as function of p N , for all sources consisting 
of four source letters. 

Theorem 2 Let S = (pi,P2,P3,P4) be a discrete source and p^ = p N be its least likely 
source letter probability. The redundancy of the corresponding Huffman code is upper 
bounded by: 



' 1 + 5p N - H{1 - Zp N ,p N ,p N ,p N ) ifO <p N < 1/9 

2 -#(1/3, 1/3, 1/3 - M J ifl/9<p N <l/6 

r<<2-H(2p N ,l-4p N ,p N ,p N ) ifl/&<P N <S 1 (12) 

2-#(^,^,^,pJ l fS 1 <p N <l/5 

2-H(l-3p N ,p N ,p N ,p N ) tfl/5<p N <l/4: 



where Si ~ 0.1708 is the unique point in the interval [1/6, l/5[ for which the function 
H(2x,l —4:X,x,x) is equal to the function H(^-^ 1 1 ~^ x r , 1 ~^ x ,x). The bound is tight. 

Proof. The Huffman code for S has length vector (2,2,2,2) or (1,2,3,3). It is easy to 
see that if the length vector is (2,2,2,2) one has pi < p 3 + £>4, otherwise pi > p 3 + Pi- 
If pi = p3 + Pi then there are two Huffman codes with the two length vectors. Now 
we distinguish between the two cases: 




.05 0.1 0. 15 0.2 0.25 0.3 



Figure 1: The upper bound on the redundancy r as function of p N for sources with 
N = 3 letters 

A) P! <p 3 + P N 

B) Pi >p 3 +P N - 

CASE A : pi < p 3 + p N . The expected codeword length is 2 and the redundancy is 
r = 2 — H(pi 7 p 2} p3 } p4). For a fixed value of p 4 = p N and considering all possible 
values for P\ } p 2} p3 we obtain that 

r<2- mm H(y 1 ,y 2 ,y 3 ,p N ) 

(yi,y2,y3,p N )sQ 

where Q = {(y 1 ,y 2 ,y 3 ,p N )\yi > IJ2 > 1J3 > p N ,yi + IJ2 + IJ3 + P N = 1, 2/i < Us + P N } is 
the set of sources with four letters whose least likely source letter has probability p N 
and that satisfies y\ < y 3 + p N . Now we compute the minimum value of the entropy. 
The computation is based on the convexity of the function H. First note that the 
minimum over Q satisfies y 2 = J/3. Indeed, since y\ < y3 + p N if y 2 > J/3 then the point 
y[ = yi + e ,2/2 = 2/2 - e,y' 3 = y 3 would satisfy H(y[,y' 2 ,y' 3 ,p N ) < H(y 1 ,y 2 ,y 3 ,P N ), for 
small e > 0. Hence y 2 = y 3 . 
It is easy to see that the function to minimize, Hil — 2y 3 — p N} y3 } y 3} p N ) } is an 
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increasing function of y 3 . Thus it assume the minimum value in the leftmost point 
in its interval of variation. From the conditions y 3 > p N and y\ < y 3 + p N one has 
ma,x{p N ,(l — 2p N )/3} < y 3 . The maximum between p N and (1 — 2p N )/3 is p N if 
p N > 1/5 and (1 — 2p N )/3 otherwise. Hence 



(2-H(l- 3p N ,p N ,p N ,p N ) if 1/5 <p N < 1/4 

r < 1 o 7T/1+P N i-2 Pjvr i-2 Pn m x ., n ^ ^ . -, ,, (13) 



ff(^,^,^,aJ ifo<^<i/5. 



CASE B : pi > p 3 + p N . Notice that p N < 1/5. Indeed if p^ > 1/5 then pi < p 3 + p N - 
The expected codeword length is 1 + p 2 + 2p 3 + 2p 4 . For a fixed value of p 4 = p N and 
considering all possible values for p\ } p 2} p 3 we obtain that 

r <l + 2p N + max {y 2 + 2y 3 - H(y 1 ,y 2 ,y 3 ,p N )} 

(yi,y2,y3,p N )sQ 

where Q = {(y 1 ,y 2 ,y 3 ,p N )\y 1 > y 2 > y 3 > P N , Vi + 1J2 + IJ3 + P N = 1, Vi > IJ3 + P N } 
is the set of sources with four letters whose least likely source letter has probability 
p N and that satisfies y\ > y 3 + p N . The function to maximize, y 2 + 2y 3 — Hil — y 2 — 
J/3 — P N iV2iyziP N )i is a convex U function of y 2 . Therefore it assumes the maximum 
at an extreme point of the variation interval of y 2 . From conditions y\ > y 2 > y 3 and 
V\ > 2/3 + p N we get y 3 < y 2 < min{l - 2y 3 - 2p N ,{\ - y 3 - p N )/2}. It is easy to see 
that 1 — 2y 3 — 2p N < (1 — y 3 — p N )/2 iff y 3 > 1/3 — p N - We distinguish the three cases 
B.l) j/2 = 1 - 2y 3 - 2p N and y 3 > 1/3 - p N 
B.2) y 2 = (1 - y 3 -p N )/2 and j/ 3 < 1/3 -^ 

B.3) J/2 = J/3- 

CASE B.l : j/i > y 3 -\-p N7 y 2 = l — 2y 3 —2p N and j/ 3 > 1/3— p N . The function to maximize 
is 1 — 2p N — H(y 3 -\-p N7 1 — 2p N — 2y 3} y 3n p N ). This function is a convex U function of y 3 
and thus it assumes the maximum value at an extreme point of its variation interval. 
From conditions y\ > y 2 > y 3 > p N one has max{p N7 1/3 — p N } < y 3 < (1 — 2p N )/3. 
The maximum between p N and 1/3 — p N is p N iff p N > 1/6. 
If j/ 3 assume the value of the leftmost point of its variation interval we get: 

f 2 - H(2p N ,l - ip N ,p N} p N ) if 1/6 <p N < 1/5 
-\2-#(l/3,l/3,l/3- M J if0<^<l/6. l > 

If j/3 = (1 - 2^)/3 then (13) holds. 

CASE B.2 : yi > y 3 + p N ,y 2 = (1 - y 3 - p N )/2 and y 3 < 1/3 - p N . Observe that 
1/3 — p N > y 3 > p N implies that p N < 1/6. The function to maximize, (1 — y 3 — 
P N )/ 2 + 2 y3-H((l -y 3 -p N )/2,(l -y 3 -p N )/2,y 3 ,p N ), is a convex U function of y 3 , 
and thus it assumes the maximum at an extreme point of its variation interval. From 



conditions y 2 > y 3 > P N and y x > y 3 + p N we get p N < y 3 < (1 - 3p N )/3. 
If V3 = P N we get 

r < 1.5 + 3p N - H((l - 2p N )/2, (1 - 2p N )/2,p N ,p N ) if < p N < 1/6. (15) 

If j/3 = (1 - 3pJ/3 then (14) holds. 

CASE B.3 : j/i > y 3 + p N and j/ 2 = J/3- The function to maximize, y 3 + 2j/ 3 — Hil — 

2 J/3 — P N ^y 3 ^y 3 ^P N )^ is a convex U function of y 3} and thus it assumes the maximum at 

an extreme point of its variation interval. From conditions y 3 > p N and y\ > y 3 -\-p N , 

one has p N < y 3 < (1 - 2p N )/3. 

If J/3 = (1 — 2p N )/3 then (15) holds, otherwise 



r <l + 5p N - H{1 -Zp N ,p N ,p N ,p N ) if <p N < 1/5. 



(16) 



Now, comparing (13)-(16) and taking the maximum we get (12) 
The bound is reached by the following sources: 

' (L-ZPniPniPniPn) ifO<p N <l/9 

(1/3, 1/3, 1/3 - M J if 1/9 < p^ < 1/6 

if 1/6 < p N < 6 X 

if <5i < P N < 1/5 
if 1/5 < ^ < 1/4. 



(2^,1 -4^,^,^) 

/ 1 +Pm 1 ~ 2 Pn 1 ~ 2 Pa 



J2L 



m. 



m. 



v 3 ' 3 ' 3 ' 

^ -3p N ,P N ,P N ,P N ) 



Pn 



This concludes the proof. 

Bound (12) is depicted in Figure 2. 



3.3 An upper bound when N=5 

In this section we consider sources consisting of five source letters and derive an upper 
bound as function of the least likely source letter probability. 

Theorem 3 Let S = (pi } p2 } p 3} p4 7 P5) be a discrete source and p 5 = p N be its least 
likely source letter probability. The redundancy of the corresponding Huffman code is 
upper bounded by: 



' 1 + 8p N - H{1 - ±p N ,p N ,p N ,p N ,p N ) 



N 1 f N 1 f N 1 f N 1 f N , 

13/6+^/2-^(1,1,^,^,^) 

'N ' Pn 1 Pn 1 Pn 1 Pn ) 

N " V 2 ' 2 ' Ln ' t J N ' t J N ) 

2 { x ~Pn) 1 ~P N 1 ~Pn ] 

5 ' 5 ' 5 ' " 

^Pn ' Pn i Pn ' Pn ' Pn ) 
( 1 ~P N 1 ~Pn 1 ~Pn 1 ~Pk 
\ 4 1 4 ' 4 ' 4 



2 + 2p N - H(3p 
r<{2 + 2p N -H{ 

ll/5 + 4 PjV /5-# 
2 + 2p N - H(l - 
[9/A + 3p N /A-H 



~Pn 



lfO<P N <$2 

ifS 2 <P N <l/9 

ifl/9<P N <l/8 

ifl/S<p N <6 3 

Pn) l f S 3 <P N < 1/6 

ifl/G<P N <8 4 
ifS 4 <p N <l/5 



;i7) 




Figure 2: The upper bound on the redundancy r as function of p N for sources with 
N = 4 letters 



where 6 2 — 0.078184 is the unique point in ]0, l/9[ /or which the function 1 + 8x 
i7(l — 4a; , x, x, x, x) «'s egwa/ to £/ie function 13/6 + x/2 — -H"(|, |, 1 ~ 3:g , 1 ~ 3:g , x); 
<5 3 ~ 0.143815 «'s £/ie unique point in ]l/8,l/6[ for which the function 2 + 2x 



-2x 1-Ax 



2 , , x, x, x) is equal to the function 6/5 + 4x/5 — H( — - - , ^r^, ^r^, ^r^, x) 

~ 0.179669 «'s £/ie unique point in ]l/6, l/5[ /or which the function 2 + 2a; — i7(l - 
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4x, x, x, x, x) «'s egwa/ to £/ie function 5/4 + 3x/4 
TTie bound is tight. 



Proof. We distinguish between four possible cases: 

A) p 2 < Pa + J>5 and p x < p 2 + p 3 

B) p 2 < Pa + P5 and pi > p 2 + J>3 

C) p 2 > pa + P5 and p x < p 3 + p 4 + J>5 

D) p 2 > Pa + Pb and p x > p 3 + p 4 + p 5 . 



CASE A : p 2 < Pa~\- Ps and pi < p 2 -\-p 3 . It is easy to see that the Huffman code has 
length vector (2,2,2,3,3) and the expected codeword length is 2 + p 4 -\-p^- Maximizing 



the redundancy over all possible values of pi 7 p 2} p 3} p4 } given that p 5 = p N7 we get 

r<2 + p N + max {y 4 - H{y u y 2 , y 3 , Va,P n )} 

(yi,y2,y3,yi,p N )sQ 

where Q = {(j/i, y 2 , y 3 , V4,P N )\yi > 2/2 > 2/3 > Va > P N , 2/i + 2/2 + 2/3 + IJa + P N = 1,2/2 < 
2/4 + PntVi < 2/2 + J/3} is the set of sources with five letters whose probabilities satisfy 
the constraints of case A. 

Observe that the maximum satisfies y 2 = J/3. Indeed, since j/i < J/2 + J/3, if J/2 > J/3 the 
point 2/2 = 2/2 + e,J/ 3 = J/3 - e,J/' = JA', z = 1,4,5 would satisfy H(y[,y 2 ,y' 3 ,y 4 ,p N ) < 
-H^J/i, J/2, J/3, JA,^), for small e > 0. Following a similar reasoning we have that 
J/3 = J/4- Hence, the function to maximize is j/ 4 — Hil — 3j/ 4 — p N , 2/4, 2/4, y4 7 p N ). This 
is a convex U function of j/ 4 , and thus it assumes the maximum at an extreme point 
of j/4's variation interval. The conditions 2/1 > 2/2, j/ 4 > p N and J/i < J/2 + J/3 imply that 
max{p N7 (l —p N )/5} < J/4 < (f —p N )/4:. The maximum between p N and (I —p N )/5 is 
£V ^ £V — f/6 an d (f —p N )/5 otherwise. Hence 

r < flI/5 + 4^/5-i7(2^,^,^,i^,pJ if < Pjv <l/6 (lg) 

I 2 + 2p N - i/(l - 4^,^,^,^,^) if 1/6 < p N < 1/5. 

H 2/4 = (1 -#J/4 we get 

r < 9/4 + 3^/4 - # (i^-, i^-, i^-, ^,P,) if < ^ < 1/5 (19) 

CASE B : p 2 < P4 + Ps and p 4 > p 2 + p 3 . Notice that p N < 1/6. Indeed, if p N > 1/6 
one would has J2Pi > 1- The Huffman code has length vector (1,3,3,3,3) and expected 
codeword length 3 — 2pi . Fixed p 5 = p N , maximizing the redundancy over all possible 
values of pi } i = I, 2, 3, 4 one has: 

r<3- min |2j/i + H{y u y 2 , j/ 3 , Va-,P n )} 

(yi,y2,y3,yi,p N )sQ 

where Q = {(j/i, J/ 2 , J/3, J/i^Jlj/i > J/2 > J/3 > J/4 > P N , J/i + J/2 + J/3 + J/4 + P N = 1, 2/2 < 
J/4 + £V> J/i — J/2 + J/3} i s the set of sources with five letters whose probabilities satisfies 
the constraints of case B. 

The minimum satisfies 2/4 = p N or y 2 = y 3 . Indeed, if j/ 4 > p N and y 2 > y 3 the 
point J/ 3 = J/3 + e, J/ 4 = J/4 — e, J/' = J/i, & = 1, 2, 5 would have a smaller entropy. We 
distinguish between the two cases 
B.I) J/4 = P N 
B.2) 2/ 2 = 2/3. 

CASE B.l (The following inequalities hold: 2/2 < 2/4 + 2/5, 2/i ^ 2/2 + 2/3 an( f 2/4 = P N )- 
A point of minimum satisfies 2/3 = 2/4 or 2/2 = 2p N . Indeed, if y 3 > j/ 4 and y 2 > 2p N 
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the point 2/3 = 2/3 + e, J/ 4 = J/4 — e, j/' = J/;, z = 1,2,5 would have a smaller entropy. 
We distinguish between the cases 

B.l.l) 2/3 = J/4 
B.1.2) j/ 2 = 2p N 

CASE B.l.l (The following inequalities hold: y 2 < J/4 + J/5, J/i > J/2 + J/3, J/4 = 2V, 
J/3 = J/4)- The function to minimize, 2(1 - y 2 - 3p N ) + H{1 - y 2 - 3p N ,y 2 ,p N ,p N ,p N ), 
is a convex D function of y 2 and it assumes the maximum at an extreme point of its 
variation interval. From the conditions y 2 > y 3} y 2 < j/ 4 + p N and J/i > J/2 + J/3 one 
has p N < J/2 < min{2p JV , (1 — 4p N )/2}. The minimum between 2p N and (1 — Ap N )/2 is 
2p N iff P N < 1/8. Hence 

< fl + 10^ -H(l -5p N ,2p N ,p N ,p N ,p N ) H0<p N <l/8 
r -\2 + 2p N -H( 1 -^, 1 -^,p N ,p N ,p N ) if 1/8 < ^ < 1/6. [ZU) 

If J/4 = p N we get 

r < 1 + % - #(1 -4^,^,^,^,^) if < p^ < 1/6. (21) 



CASE B.1.2 (The following inequalities hold: y 2 < j/ 4 + J/5, J/i > J/2 + J/3, J/4 = JW, J/2 = 
2p N ). Notice that p N < 1/8. Indeed if p^ 1/8 one would have Y,Pi > 1- The function 
to minimize, 2(1-4^ -y 3 ) + H(l-4:p N -y 3l 2p N1 y 3l p N1 p N ) 1 is a convex n function of 
y 3 and it assumes the maximum at an extreme point of j/ 3 's variation interval. From 
the conditions J/i > j/ 2 + J/3, J/2 > J/3 > J/4 we get p N < y 3 < rain{2p N , (1 - 6p N )/2}. 
The minimum between 2p N and (1 — 6p N )/2 is 2p N iff p^ < 1/10. Hence 

< J 1 + 6^ -#(1-6^,2^,2^,^,^) if < p^ < 1/10 

r -\ 2 + 2^-#(i^,2^,^, M J if 1/10 < p N < 1/8. [ZZ) 

If J/3 = ^ (20) holds. 

CASE B.2 (The following inequalities hold: y 2 < J/4 + J/5, J/i > J/2 + J/3 and j/ 2 = J/3)- 

The function to minimize, 2j/i + H(yi, — yi ^ Pjv , — yi ^ Pjv , j/4,p w ), is a convex fl 

function of j/ 4 . Thus, it assumes the maximum value at an extreme point of j/ 4 's 

variation interval. From conditions j/i > y 2} y 3 > J/4 > p N , and j/ 2 < J/4 + P N one 

has maxjp^, (1 — j/i — 3p N )/3} < j/4 < 1 — 2j/i — p N . The maximum between p N and 

(1 — j/i — 3p N )/3 is p N iff p N > (1 — j/i)/6. We distinguish between the three cases 

8.2.1)2/4 = 1-2?/!-^ 

B.2. 2) j/4 = p N and ^ > (1 - j/i)/6 

B.2. 3) j/4 = (1 - j/i - 3p N )/3 and ^ < (1 - j/i)/6. 

CASE B.2.1 (The following inequalities hold: J/2 < J/4 + J/5, J/i > J/2 + J/3, J/2 = J/3, 
j/4 = 1 - 2j/i -fly). The function to minimize, 2y x + H(y 1 ,y 1 /2, j/i/2, 1 - 2j/ x -p N ,p N ), 
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is a convex D function of j/ 4 and, thus, it assumes the maximum value at an extreme 
point of j/i's variation interval. From condition j/ 4 > j/ 2 + J/3, J/3 > J/4 > _p N and 
J/2 < J/4 + J/5 one has 2(1 — p N )/5 < j/i < min{(l — 2p N )/2 } 2/5}. The minimum 
between (1 - 2pJ/2 and 2/5 is 2/5 iff ^ < 1/10. Hence 

J 11/5 - #(2/5, 1/5, 1/5, 1/5 - Pn , Pn ) H0< Pn < 1/10 
-l2 + 2^-#(i^,i^,i^, M J if 1/10 < 1%. < 1/6. {Z6) 

If yi = 2 J1^m1 then (18) holds. 

CASE B.2.2 (The following inequalities hold: J/2 < J/4 + J/5, J/i > J/2 + J/3, J/2 = J/3, 
J/4 = £V an d £V — (1 ~~ J/i)/6)- The function to minimize 2(1 — 2j/ 3 — 2p N ) + #(1 — 2j/ 3 — 
2p N , J/3, J/3,£V,£v) is a convex fl function of j/ 3 and thus, assumes the maximum at an 
extreme point of its variation interval. From condition J/i > J/2 + J/3, J/3 > J/4, J/2 < 
j/4 + p N and p N > (1 - j/i)/6 one gets p N < y 3 < rain{2p N , (1 - 2p N )/4}. One has 
2p N < (1 - 2pJ/4 iff p N < 1/10. If j/3 = 2p N we get 

r < 1 + l2p N - H(l - 6p N} 2p N} 2p N} p N} p N ) if < p N < 1/10. 

(24) 

If j/3 = p N or j/3 = (1 — 2p N )/4: then respectively (21), (23) hold. 

CASE B.2.3 (The following inequalities hold: J/2 < J/4 + J/5, J/i > J/2 + J/3, J/2 = J/3, 
j/4 = (1 — j/i — 3p N )/3 and p N < (1 — j/i)/6). The function to minimize, 2j/i + #(j/i, (1 — 
j/i)/3, (1 — J/i)/3, (1 — j/i — 3p JV )/3,p JV ), is a convex fl function of j/i. From condition 
J/i > J/2 + J/3, J/4 > JW and £W < (! _ 2/i)/ 6 one S ets 2/5 < j/i < 1 - 6p N . In these 
extreme points (23) and (24) hold. 

CASE C : P2 > Pa + Ps and p 4 < p 3 + p 4 + p 5 . Notice that p N < 1/7. Indeed, if 
p N > 1/7 one would have ^p,- > 1. The Huffman code has length vector (2,2,2,3,3) 
and the expected codeword length is 2 + p 4 + p 5 . like previous cases we hx the least 
probability p 5 = p N7 and consider all possible values of pi 7 i = 1,2,3,4, obtaining an 
upper bound for the redundancy. 

r<2 + p N + max {j/ 4 - H{y u j/ 2 , j/ 3 , y A ,p N )} 

(yi,y2,y3,yi,p N )sQ 

where Q = {(j/i, J/ 2 , J/3, J/4,P N )|j/i > J/2 > J/3 > J/4 > P N , J/i + J/2 + J/3 + J/4 + P N = l,J/ 2 > 
J/4 + £V> 2/i < J/3 + J/4 + Bvl i s ^ e se ^ °f sources with five letters whose probabilities 
satisfies the constraints of this case. 

Observe that a point of maximum satisfies j/i = j/ 2 or j/ 3 = j/ 4 . Indeed, if j/ 4 > j/ 2 
and j/3 > j/ 4 then point J/ 2 = J/2 + e, J/3 = J/3 — e, J/' = J/i, & = 1,4,5 would satisfy 
H(y'i,y' 2 ,y3,y4,P N ) < #(j/i,J/2,J/3,J/4,Jw), for small e > 0. We distinguish between the 
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two cases 

CI) J/1 = J/2 
C2) J/3 = J/4- 

CASE C.l (The following inequalities hold: j/ 2 > y 4 + p N , J/i < J/3 + J/4 + p N and 

\ t-u t +• + • • u( 1 -y3-yi-p N i-y3-yi-p N \ ■ 

J/i = J/2J- Hie tunction to maximize, y 4 — n{ ^ — , ^ — , j/ 3 , j/4,^), is 

a decreasing function of j/ 3 , and assumes the maximum value at leftmost point of 

j/3's variation interval. From conditions j/ 3 > y 4 and j/i < j/ 3 + j/ 4 + p N one has 

max{j/ 4 ,l/3-j/ 4 -Jw} < J/3- It is easy to see that y 4 > l/3-y 4 -p N iff y 4 > (l-3p N )/6. 

We distinguish between the two cases 

CI.I) J/3 = J/4 and J/4 > (1 - 3p N )/6 

CI. 2) J/3 = 1/3 -y 4 -p N and j/ 4 < (1 - 3#J/6. 

CASE C.l.l (The following inequalities hold: j/ 2 > j/ 4 + p N , J/i < J/3 + J/4 +P N , J/i = J/2, 

j/3 = j/4 and j/4 > (1-3^ )/6). The function to maximize, y 4 -H( ^— ^-, ^-^ IL ,y 4 ,y 4 ,p N ), 

is a convex U function of j/ 4 , thus assumes the maximum at an extreme point of its 
interval of variation. From conditions j/ 4 > p N , y 2 > J/4 + p N7 yi < J/3 + J/4 + P N and 
j/4 > (1 - 3p N )/6 one gets max{p N , (1 - 3p N )/6} < j/ 4 < (1 - 3#J/4. p N is greater 
than (1 — 3p N )/6 iff p N < 1/9. Hence 



f 13/6 + p,/2 - #(§, |, i^, ^l,^) if < p N < 1/9 
1 2 + 2p N - H{±^l, 1 -^,p N ,p N ,p N ) if 1/9 < p N < 1/7. 



If j/4 = (l-3 PjV )/4 then 

r<9/4+^/4-#(i±^,i±^,i^,i^, P J if0<^<l/7. (26) 

CASE C.l. 2 (The following inequalities hold: j/ 4 + p N , j/i < J/3 + J/4 + 2V, J/i = J/2, 
J/3 = 1/3 - j/4 - JW and J/4 < (1 - 3p JV )/6). Observe that p N < j/ 4 < (1 - 3p N )/6 
implies^ < 1/9. The function to maximize, j/ 4 — #(1/3, 1/3, 1/3 — j/4 — p N7 y 4} p N ) } 
is a convex U function of j/ 4 , and assumes the greatest value at an extreme point of 
its variation interval. From conditions j/ 3 > y 4 > p N one gets p N < j/ 4 < (1 — 3p N )/6. 
Hence 

r < 2 + 2 PjV - #(1/3, 1/3, (1 - 6p)/3, M J if < p N < 1/9. (27) 

If j/4 = (1 - 3pJ/6 then (25) holds. 

CASE C.2 (The following inequalities hold: j/ 2 > y 4 + p N , j/i < J/3 + J/4 + p N and 
J/3 = J/4)- Since J/i < J/3 + J/4 + p N and j/ 2 > j/4 + p N a point in this case can not be 
of maximum. Indeed the point y[ = yi + e, J/ 2 = J/2 — e, J/; = J/i, i = 3, 4, would have 
smaller entropy. 
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CASE D : p 2 > p 4 + Ps and pi > p 3 + p 4 + p N . Notice that p N < 1/8. Indeed if 
p N > 1/8 one would has Y,Pi > 1- The Huffman code has length vector (1,2,3,4,4). 
Like the previous cases we upper bound the redundancy by 

r<l + 3p N + / max {y 2 + 2j/ 3 + 3j/ 4 - #(j/i, j/ 2 , J/3, J/4,2v)} 

{yi,y2,ys,y4,p N )sQ 

where Q = {(j/i, y 2 , y 3 , y 4 ,P N )\yi > J/2 > J/3 > J/4 >P N ,Vi + J/2 + J/3 + J/4 + P N = l,j/ 2 > 

J/4 + P N i J/i — J/3 + J/4 + Pn} * s ^ e se ^ °f sources with hve letters whose probabilities 

satisfies the constraints of case D. 

The function to maximize, y 2 + 2y 3 + 3j/ 4 - H(l - y 2 - y 3 - y 4 - p N ,y 2 , y 3 , y 4 ,p N ), 

is a convex U function of y 2 and thus assumes the maximum at an extreme point of 

j/2 's variation interval. From conditions y\ > y 2 > y 2 + p N , y\ > y 3 + y 4 + p N we 

get y 4 + p N < y 2 < min{ ys yi Pn , 1 - 2y 3 - 2y 4 - 2p N }. Moreover ys yi Pn < 

1 — 2y 3 — 2y 4 — 2p N iff y 3 + y 4 + p N < 1/3. We distinguish between the three cases 

D-l) J/2 = y 4 + p N 

D.2) j/2 = 1 - y3 -^ and y 3 + y 4 + p N < 1/3 

D.3) y 2 = 1 - 2y 3 - 2y 4 - 2p N and y 3 + y 4 + p N > 1/3. 

CASE D . 1 (The following inequalities hold: y 2 > y 4 +y 5 , J/i > J/3+J/4+P N , J/2 = V4+P N )- 
The function to maximize, (y 4 +p N ) + 2y 3 + 3y 4 -H(l-y 3 -2y 4 -2p N ,y 4 +p N ,y 3 ,y 4 ,p N ), 
is a convex U function of y 3 and assumes the greatest value at an extreme point of 
j/ 3 's variation interval. From conditions y\ > y 3 + y 4 + p N , y 2 > y 3 > y 4 one gets 

J/4 < J/3 < mm{y 4 +p N , y \ Pn }. Furthermore y 4 +p N < y \ Pn iff y 4 +p N < 1/5. 

We distinguish the cases 

D.l.l) y 3 = y 4 

B.l.2)y 3 = y 4 +p N <l/5 

D.1.3) j/3 = ' ^ 3Pw and j/ 4 +P N > 1/5. 

CASE D.l.l (The following inequalities hold: y 2 > y 4 + y 5 , y\ > j/ 3 + j/ 4 + p N , j/ 2 = 

J/ 4 + p N , J/3 = J/4)- The function to maximize, y 4 +p N +2y 4 + 3y 4 -H (1 -3y 4 -2p N , y 4 + 

p N7 y 4} y 4} p N ) } is a convex U function of y 4 and assumes the maximum at an extreme 

point of its variation interval. By conditions j/ 4 > y 2 > y 4 + p N and j/ 4 > y 3 + j/ 4 + p N 

one gets p N < y 4 < (1 — 3p N )/5. Hence 

r< l + 10^-#(l-5^,2 Mw , M JifO<^<l/8 (28) 

r< 11/5 + 2^/5 -/T(^,i±^,i=|aL,l=^L,^) if 0<^< 1/8. (29) 

CASE D.1.2 (The following inequalities hold: y 2 > y 4 + y 5 , j/ 4 > y 3 + j/ 4 + p N , j/ 2 = 
j/4 + p N , J/3 = J/4 + P N < 1/5). The condition y 4 + p N < 1/5 together p N < y 4 implies 
that p N < 1/10. The function to maximize, y 4 + p N + 2(j/ 4 + p N ) + 3j/ 4 — i7(l — 3j/ 4 — 
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3p N7 y 4 -\-p N7 y 4 -\-p N7 y 4} p N ) } is a convex U function of y 4 and assume the maximum at an 
extreme point of its variation interval. From conditions p N < y 4 and j/i > j/ 3 + J/4 + £V 
one gets p N < j/ 4 < (1 — 5p N )/5. If j/4 = p N then (24) holds. If j/ 4 = (1 — 5p N )/5 then 
(23) holds. 

CASE D.1.3 (The following inequalities hold: y 2 > y 4 + J/5 , J/i > J/3 + J/4 + p N , J/2 = 

J/4 + Bv5 J/3 = V ~\ — ~ an( i J/4 + Pn > 1/5)- The function to maximize, y 4 + p N + 

1 - 3j/ 4 - 3p N + 3p N - ff( ^ % ,j/ 4 + JW, ^ 2 % ,j/ 4 ,iv), is a convex U function 

of J/4 and assumes the maximum at an extreme point of its variation interval. By 
conditions y 2 > y 3 > J/4 > P N one gets max{p N ,(l - 5p N )/5} < y 4 < (1 - 3p N )/5. 
Moreover, p N is greater than (1 — 5p N )/5 iff p N > 1/10. If y 4 = (1 — 5p N )/5 then (23) 
holds, if J/4 = (1 — 3p N )/5 then (29) holds, and if y 4 = p N then (22) holds. 



CASE D.2 (The following inequalities hold: y 2 > J/4 + J/5, J/i > J/3 + J/4 + p N , J/2 = 
w and y 3 -\-y 4 -\-p N < 1/3). Observe that, since J/3 + J/4+IV < 1/3 one has p N < 



1/9. The function to maximize, 1 ~ y3 ~% i ~ p » +2y 3 +3y 4 -H( 1 ~ y3 ~![ i ~ p " , '"^"f"^ , j/ 3 , J^,^ 
is a convex U function of j/ 3 and assumes the maximum at an extreme point of 
its variation interval. From conditions j/ 3 > y 4 and j/i > j/ 3 + y 4 + p N one gets 
j/4 < j/3 < 1/3 — y 4 — p N . We distinguish the two cases 
D.2.1) J/3 = J/4 

D.2.2) j/3 = 1/3-j/4-^. 

CASE D.2.1 (The following inequalities hold: j/ 2 > y 4 + j/5 , j/i > y 3 + j/4 + p N , j/2 = 

— ^ ^ Pjv ,j/ 3 + J/4 + p N < 1/3 and j/ 3 = j/ 4 ). The function to maximize, Vi Pn + 

2j/ 4 + 3j/ 4 - #((1 - 2j/ 4 — p Jv .)/2, (1 - 2j/ 4 - p N )/2,y 4 ,y 4 ,p N ), is a convex U function 
of j/4 and assumes the maximum at an extreme point of its variation interval. By 
conditions y 4 > p N and J/i > J/3 + J/4 + p N we have p N < j/ 4 < (1 — 3p N )/6. Hence, if 
j/4 = p N then 

r < 3/2 + 13p N /2 - H (^|^, X -^ , p N , p N , p N ^j if < p N < 1/9. (30) 

If j/4 = (1 - 3p N )/6 then 

r < 2 + 2p N - #(1/3, 1/3, (1 - 3pJ/6, (1 - 3pJ/6,pJ if < p N < 1/9. (31) 



CASE D.2.2 (The following inequalities hold: j/ 2 > y 4 + j/5 , j/i > j/3 + j/4 + p N , j/2 = 

^— ^ — — ,J/3 + J/4 + P N < 1/3 and j/ 3 = 1/3 — y 4 — p N ). The function to maximize, 

1/3 + |(1 — 3j/4 — 3p N ) + 3j/4 — #(1/3,1/3, 1 — —,y 4 ,p N ), is a convex U function 

of j/4 and assumes the maximum at an extreme point of its variation interval. By 
conditions j/ 3 > y 4 > p N one gets p N < j/ 4 < (1 — 3p N )/6. Hence If j/ 4 = (1 — 3p N )/6 
then (31) holds and if y 4 = p N then 

r<2 + 2p N - #(1/3, 1/3, (1 - 6 P J/3, M J if < Pjv < 1/9. (32) 
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CASE D.3 (The following inequalities hold: j/ 2 > J/4 + J/5, J/i > J/3 + J/4 + p N , J/2 = 
1 — 2j/ 3 — 2j/ 4 — 2p N and j/ 3 + j/ 4 + p N > 1/3). The function to maximize, (1 — 2j/ 3 — 
2j/ 4 ) + 2j/ 3 + 3j/ 4 - #(j/ 3 + J/4 + P N , 1 - 2j/ 3 - 2j/ 4 , j/3, y 4 ,p N ), is a convex U function 
of j/3 and assumes the maximum at an extreme point of its variation interval. From 

conditions j/i > j/ 2 > j/3 > j/4 and j/ 2 > y 4 + P N one has that max{ ^ — — ,2/4} < 

j/4 < min{ x " 2;/4 3 - 2 ^ , i- 3 ^- 3 ^ }. It easy to see that 1 " 3 ^- 3 ^ > j/ 4 iff j/ 4 < (l-3pJ/6 
and 1 — — < 1 — - iff j/ 4 < (1 — 5p N )/5. We distinguish among the four cases 

D.3.1) j/3 = '-^-^ and j/ 4 < (1 - 3p N )/6 
D.3. 2) j/3 = j/4 and j/ 4 > (1 - 3p N )/6 
D.3.3) j/3 = i^^ and j/ 4 < (1 - 5pJ/5 
D.3.4) j/3 = 1-3;/4 2 ~ 3% and j/ 4 > (1 - 5pJ/5. 

CASE D.3.1 (The following inequalities hold: J/2 > J/4 + J/5 , J/i > J/3 + J/4 + £V , J/2 = 
1 - 2j/ 3 - 2j/ 4 - 2p N , y 3 + V4+P N > 1/3, j/3 = ~ m 3 ~ Pn and j/ 4 < (1 - 3p N )/6). By 
condition p N < j/ 4 < (1 — 3p N )/6 one has p N < 1/9. The function to maximize , 
1/3 + |(1 — 3j/ 4 — 3p N ) + 3j/ 4 — i7(l/3,l/3, — ^ — —,y4,p N ), is a convex U function 
of j/ 4 and assumes the maximum at an extreme point of its variation interval. From 
conditions of this case we get p N < j/ 4 < (1 — 3p N )/6. In both extreme point a bound 
already obtained holds. Precisely (31) and (32) hold. 

CASE D.3.2 (The following inequalities hold: J/2 > J/4 + J/5 , J/i > J/3 + J/4 + £V , J/2 = 
l-2y 3 -2y 4 -2p N , J/3 + J/4 + JW > V 3 , J/3 = J/4 and j/ 4 > (l-3p JV )/6). The function to 
maximize, (l-4j/ 4 -2p JV ) + 2j/ 4 + 3j/ 4 - J ff(2j/ 4 +p JV , l-iy 4 -2p N , j/ 4 , y 4 ,p N ), is a convex U 
function of j/ 4 and assumes the maximum at an extreme point of its variation interval. 
From conditions of this case we get max{p w , (1 — 3p N )/6} < j/4 < (1 — 3p N )/5. The 
maximum between p N and (1 — 3p N )/6 is p N iff p N > 1/9. Hence 



r < 2 + 2p- H(3p N ,l - 6p N ,p N ,p N ,p N ) if 1/9 <p N < 1/8. (33) 

Whereas if j/ 4 = (1 — 3p N )/6 or j/ 4 = (1 — 3p N )/5 then (31) and (29) respectively holds. 
CASE D.3.3 (The following inequalities hold: J/2 > J/4 + J/5 , J/i > J/3 + J/4 + £V , J/2 = 
1 - 2j/ 3 - 2j/ 4 - 2p N , y 3 + V4+P N > 1/3, j/3 = ~ " 3" % and j/ 4 < (1 - 5p N )/5). The 
condition p N < j/ 4 < (1 — 5p N )/5 implies that p N < 1/10. The function to maximize, 
1-2^2^ + 2(i-2y,-2 PN ) +3 ^ _ #(111/4+^ 1-2^-2^ i-2 yt -2p N ^ p ^ is a convex U 

function of j/ 4 and assumes the maximum at an extreme point of its variation interval. 
From conditions of this case we get p N < j/ 4 < (1 — 5p N )/5. Hence 

r <2 + 2p N - H (^^,^^,^^,p N , Pl }j if 1/9 < ^ < 1/10. (34) 

If j/4 = (1 - 5pJ/5 then (23) holds. 
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CASE D.3.4 (The following inequalities hold: y 2 > y 4 + J/5 , j/i > J/3 + y 4 +p N , J/2 = 1 — 
2y 3 -2y 4 -2p N , y 3 + y 4 + p N > 1/3, j/ 3 = x ~ 3;/4 2 ~ 3 ^ and j/ 4 > (l-5p N )/5). The function 
to maximize , y 4 +p N + 1 - 3y 4 - 3p N + 3y 4 - H( 1 ~ yi ~ PN , y 4 + p N} 1 ~ 3yi ~ 3p ^ ; y 4 ,^), is a 
convex U function of j/ 4 and assumes the maximum at an extreme point of its variation 
interval. From conditions of this case we get max{p K , (1 — 5p N )/5} < y 4 < (1 — 3p N )/5. 
Moreover p N > (1 - 5p N )/5 iff p N > 1/10. If y 4 = p N , (1 - 5p N )/5, (1 - 3p N )/5 then 
respectively (22), (23), (31) holds. 

Comparing (18)-(34) and taking the maximum we get (17). 
The bound is reached by the following sources: 

(1/3, 1/3, (1 - 3p N )/6, (1 - 3 Pn )/6 }Pn ) if 6 1<Pn < 1/9 

(3^,1-6^,^,^,^) if 1/9 <p N < 1/8 

((1-2 p J/2,(1-4 p J/2, Mn , p J if l/8<^<£ 2 

(2(l-pJ/5,(l-pJ/5,(l-pJ/5,(l-pJ/5,pJ if <5 2 <^<l/6 

(L-^PniPniPniPniPn) if 1/6 < p N < <5 3 

((1 - pJ/4, (1 - p N )/i, (1 - p N )/i, (1 - pj/4,pj if 6 3 < p N < 1/5. 

This concludes the proof. □ 

Bound (17) is depicted in Figure 3. 

A simple but tedious calculus shows that the functions that represents the bound 
decreases as N increases, for N < 5 and for any fixed value of the least likely source 
letter probability. More formally, we have the following 

Lemma 4 Let a(p N ) } f3(p N ) and^(p N ) be the functions that represent the bounds (11), 
(12), and (17) respectively, i.e. the bounds for N = 3,4, 5. Then, 

a(p N ) > f3(p N ) for 0<p N < 1/4 
/3{p N ) > i(p N ) for 0<p N <l/5. 

Figure 4 shows bounds (1), (11), (12), and (17). 

4 An upper bound when only p N is known and N > 

3 

The bound r < 1 — 7i(p N ) is achieved by sources of two letters. In this section we 
prove that whenever N > 3 a sharper bound holds. More precisely, we prove that 
bound (3) holds whenever N > 3. 
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Figure 3: The upper bound on the redundancy r as function of p N for sources with 
N = 5 letters 

Theorem 4 Let S be a source with N > 3 letters and p N its least likely source letter. 
The redundancy of the corresponding Huffman code is upper bounded by: 



r < 



1 - H{2p N 



if0<P N <$ 



(35) 



0.5 + l.5p N -H(p N ), tf 6 <p N < 1/3 

where 6 ~ 0.1525 is the unique zero of the function 0.5 — 1.5a; — 7i(2x) + 7i(x) in the 
interval [0,0.3]. The bound is tight. 

Proof. By Lemma 4 the redundancy satisfies (35), for N = 3, 4, 5. Hence we assume 

N > 6 and thus p N < 1/6. 

We distinguish between the two cases p N < 1/7 and 1/7 < p N < 1/6. 

CASE A : p N < 1/7. We have to prove that r < 1 — 7i(2p N ). Now we distin- 
guish among few possible cases according to the value of the most likely source letter 
probability p^. 

CASE A.l : < pi < 1/6. From (5) we have r < pi + 0.0860 — p N which is < 
1/6 + 0.0860 - p N < 0.253 - p N < 1 - H(2p N ). 
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Figure 4: Upper bounds on the redundancy r as functions of p N for sources with 
N = 2,3,4,5 letters 



CASE A.2 : 1/6 < & < 0.1971. From (6) we have that r < 2 - 1.3219(1 - p t ) - 
7i(pi) — p N . The right-hand side is a decreasing function of pi, and thus r < 2 — 
1.3219(1 - 1/6) - ft(l/6) -p N < 0.249 - p N < 1 - ft(2pj. 

CASE A.3 : 0.1971 <pi < 0.2. From (7) we have that r < 4-18.6096pi -H(5pi)-p N . 
The right-hand side is a decreasing function of pi, and thus r < 4 — (18.6096)(0.2) — 
W(l)-^< 0.278 -^<1-^(2 P J. 

CASE A.4 : 0.2 <pi < 0.3138. From (8) we have that r < 2-1.25(l-p 1 )-H(p 1 )-p N . 
The right-hand side is an increasing function of pi, and thus r < 2 — 1.25(1 — 0.2) — 
ft(0.2) -p N < 0.278 -^ < 1 -H(2p N ). 

CASE A.5 : 0.3138 < p x < 1/3. From (9) we have that r < 3 - 3(1 + log3)pi - 
7i(3pi) — p N . The right-hand side is a decreasing function of pi, and thus r < 3 — 
3(l+log3)(l/3)-'H(l)-p JV < 0.415-^. It is easy to see that 0.415-^ < 1-^(2^) 
for p N < 0.0884. Now assume p N > 0.0884. Let t be the number of leaves at level 2 
in the Huffman code tree for the source S = (pi } . . . ,p N ). Since N > 6 then t < 3. 
As stated in [3] (see Theorem 3, equation (35)), the redundancy can be upper bounded 
by r < I - H(q", ...,q'^) + q 2m -i ~ 



where / is a level for which the code tree i 



is 
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complete, and m is the smallest integer such that node 2m — 1 with probability 
q 2m -i i n the sibling list is at level / + 1, and q", i = 1, . . . , 2 l are the entries in the 
sibling list at level /. We consider the case 1 = 2. For t = 1,2 ,Capocelli and De 
Santis [2] proved that 2 — H(q", . . . } q 2 i) + <?2m-i < 0.75 + l.25p\ — 7i(pi). Hence 
r < 0.75 + 1.25pi - ft(pi) - ^ < 0.278 - p N < 1 - ft(2 PjV ) for PjV < 1/7. 
Suppose t = 3. Since iV > 6, p N > 0.0884 and 0.3138 < p\ < 1/3, recalling 
(4), one has that an optimal code for S must have length vector (2,2,2,3,4,4). Let 
S = (pi, . . . iP&) be a source of 6 letters. There are two possible situations in which 
Huffman procedure generates a code whose length vector is (2,2,2,3,4,4): 

(a) p 3 > p 5 + p 6 , p 2 > p 4 + P5 + Pe, and p t < p 3 + p 4 + p 5 + p 6 

(b) P3> P5+ Pe, P2 < P4 + P5 + Pe, and p t < p 2 + p 3 - 

Case (a) is not possible: indeed, since p 6 > 0.0884 and pi > 0.3138, we would have 
12 Pi >P1 + {P4+P5+P6) + {P5+P6)+P4+P5+P6 > Pi + 8p 6 > 0.3138 + 8(0.0884) > 1. 
In case (b), nodes with probabilities p 2 and p 3 are merged at level 2 of the code tree. 
Observe that p 2 + p 3 > 2p 3 > 2(p 5 + p e ) > 4p 6 > 1/3, and since pi < 1/3 then 
Pi < P2 + p-i- The redundancy is r = 2 + p 4 + 2p 5 + 2p 6 - H(p 1 ,p 2 ,p 3 ,p4,p 5 ,p 6 ). It 
is well known that H(p x + e } p 2 - e } p 3} p 4} p 5} p 6 ) > H(p 1} p 2} p 3} p 4} p 5} p 6 ) for small 
e (that is, for e < min{p 2 — £>3, 1 — Pi})- Since p 4 < p 2 + p 3} if p 2 > p 3 then the 
source (p 4 + e } p 2 — e } p 3} p4 7 p 5} p 6 ) would have a redundancy greater than that of the 
source (pi } p 2} p 3} p4 7 p 5} p 6 ). Thus the maximum redundancy, for all sources satisfying 
P3 ^ Ps + p&t P2 < P4 + Ps + p&t and pi < p 2 + p 3} for a hxed value of p 6} is reached 
when p 2 = p 3 . On other hand, if p 2 = p 3 it is easy to see that the redundancy r of S 
is equal to the redundancy r' of the source of five letters (p 2 + p 3} pi 7 p 2} p 3} p4). From 
Theorem 3 and Lemma 4 we have r' < 1 — 7i(2p N ). 

CASE A.6 : 1/3 < pi < 0.4505. From (10) we have that r < 1 + 0.5(1 - p 4 ) - H(pi) - 
2p N . The right-hand side is a decreasing function of pi, and thus r< 1 + 0.5(1 — 
1/3) - H(l/3) - 2p N < 0.415 -2p N <l- H(2p N ) for p N < 1/7. 

CASE A. 7 : 0.4505 < p\. Let B be the source obtained by deleting the hrst letter in S 
and normalizing the remaining probabilities, i.e. B = (p 2 /(l — pi) } . . . } p N /(l — Pi))- 
Let L and Lb be the expected codeword lengths of sources S and B } and H and Hb 
their entropies. It is well known that 



and 

Using bound (1), we have 





H = 


n(Pi) + 


(I-Pi)Hb 




L 


= i + (i 


- Pi)L B - 


L- 


-H 






1 - 


-H( Pl ) + (l- 


Pi)tb 
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< i-n(p 1 ) + (i-p 1 : 

= 2 - Pl - H{ Pl ) - {I - Pl )H 

= 2-p!- H{p 1 ,l-p 1 -p N ,p N/ 

The function 2 — x — H(x } 1 — x — p N7 p N ) is a convex U function of x and thus it 
assumes the maximum value at an extreme point of x's variation interval. Since 
N > 6 one has 0.4505 < x < 1 — 5p N < 1 — 2p N . If x = 1 — 2p N we have that 
2 - (1 - 2p N ) - H(l - 2 Pn}Pn}Pn ) < 1 - H(2p N ). lfx = 0.4505 we have 




r < 2 - 0.4505 - ^(0.4505) - (0.5495)^ ( - 



Pn 



V0.5495, 
which by a simple calculus is less than 1 — 7i(2p N ) for p N < 1/7. 

CASE B : 1/7 < p N < 1/6. Since p N > 1/7 and N > 6 then N = 6. Thus 
1/6 < pi < 1 — 5p N = 0.28. By Lemma 2, we have that, for 1/6 < p\ < 0.28, the 
minimum codeword length of the Huffman code satisfies either rt\ = 2 or rt\ = 3. 
If rii = 3 then N would be > 8, therefore rt\ = 2. Recalling (4) we have that the 
Huffman code has length vector (2,2,3,3,3,3). Observe that if pi = p 2 or p 3 = p 4 then 
the redundancy of source S is equal to the redundancy of a source of five letters. 
From Theorem 3 and Lemma 4, (35) holds. Hence assume pi > p 2 and p 3 > p 4 . The 
redundancy is equal to r = 2 + p 3 + p 4 + p 5 + p 6 — H(pi } p 2} p3 } p4 } p5 } p6)- It is well 
known that H(p 1} p 2} p 3} P4 + e,J>5 - e,p 6 ) > H(p 1} p 2} p 3} p 4} p 5} p 6 ) } for small e > 0. 
Since p 3 > p 4} if p 5 > p 6 then the source (pi 7 p 2} p 3} p 4 + e } p 5 — e } p 6 ) would have a 
redundancy r' greater than the redundancy r of the source S. Thus a source with 
maximum redundancy, among all sources satisfying the constraints of this case, for 
a fixed value of the least likely source letter probability p 6} has p 5 = p e . Hence the 
maximum redundancy is equal to the redundancy of a source consisting of five letters 
whose least likely source letter probability is p 4 . From Lemma 4 the redundancy of a 
source of five letters whose least likely source letter probability is p 4} is less than the 
redundancy of a source of three letters whose least likely source letter probability is 
p 4 . On the other hand p 4 < (1 — p 5 — p&) / ^ < (1 — 2/7)/4 < 0.18. Thus, recalling 
that p N < p 4} from Lemma 3 we have that a(p N ) > a(p 4 ) > r. 

D 



5 An upper bound as function of p and p N 

In this section, exploiting the bound provided in Theorem 4, we obtain an upper 
bound as function of p N _ x and p N which improve Yeung's bound (2). 
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Theorem 4 tells us that r < ct(p N ) where, recalling the dehnition of a, 

a(n \ = h^ if < Pjv < ^ 
(Pn> \Hp n ) if * < fW < 1/3 

and </>(:c) = 1 — 7i(2x) and ^(x) = 0.5 + 1.5 — 7i(x). 

The two special cases N = 2 and iV = 3 are trivial since we know all probabilities of 

the source, and we can compute the exact value of the redundancy. We have 

r = l-H(p N _ 1 ) = l-H(p N )itN = 2 

r = 1 + P N _, + P N - H(l -p N _, -PhiPn-hPn) if N = 3 

Thus, we suppose that N > 4. Let A = (pi, . . . } p N _ 17 p N ) be a source with N letters 
and let B the source obtained merging p N _ x and p N , i.e. with source letter probabilities 
{pi, . . . } p N _ 27 p N _ 1 -\~P N }- The redundancies r and re of sources A and i? respectively, 
satisfy 

r = r B + l(p N _ 1 ,p N ) (36) 



where /fe^,^) = fe^ +P N ) + 



1 - W — Ss 



is the contribute to the redun- 



dancy due to p N _ x and p N . Since A has at least four letters, B has at least three 
letters and thus we can upper bound r B using Theorem 4. We distinguish between 
the two cases p N _ x + p N < p N _ 2 and p N _ x + p N > p N _ 2 . 

Consider the case p N _ x + p N < P N _ 2 • The least likely source letter probability of B 
is £V-i +£V- Since B has at least three letters, we have p N _ x + p N < 1/3. By Theorem 
4, re is upper bounded by 

r < / ^ + #v-i )> if ° < ^ + Pn-i ^ * / 37 \ 

B ~ 1 V'fe +2V-J, if<^ <2V +Pat_i < V 3 

Consider the case p JV _ 1 + p N > p N _ 2 . The least likely source letter probability of B 
is p N _ 2 and it satisfies 

P N . t < P N - 2 < minll/3,^^ +p N }. 

By Theorem 4 we have that 

r B < max a(x). 

P N -! <a:<niiii{l/3,p jv _ 1 +p jv } 

By Lemma 3, if p JV _ 1 < 8 then o;(p JV _ 1 ) is the maximumofa(i) in [p N _ 17 min{l/3,p JV _ 1 + 
p N }] and thus 

rB^^.Jifp^ <£ (38) 
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Assume p N _ x > S.We distinguish the two cases p N _ x + p N < 1/3 and p N _ x + p N > 1/3. 
In the case p N _ t + p N < 1/3, by the convexity of ip we have 

r B < maxjV'fe + ^.J, V'fe.J} if P N -! > 8 and p N-1 + p N < 1/3 (39) 

whereas, in the case p N _ x + p N > 1/3 we have 

r B < max{'tjj(l/3),'ijj(p N _ 1 )} if p N-1 > 8 and p N-1 + p N > 1/3. (40) 

Observe that p N _ x -\- p N > 1/3 implies p N _ x > 8. 

Comparing (37)- (40) and taking the maximum we obtain the following upper 
bound on re, 

' max{(f)(p N +p N _ 1 ),(f){p N _ 1 )} ifftr.j +P N < 8 

maxjV'fe +P N _ 1 ),<P(p N _ 1 )} Hp N _ t + P N > 8,p N _ l < 8 

max{V>fe +P n _ 1 )^KPn-i)} ifp N -i > S ,P N _ 1 + P N < 1/3 

,max{V'(p JV _ 1 ),^(l/3)} ifp N -i + Pn > V 3 

Substituting above bound in (36), and observing that ma,x{<f)(p N + p N _ 1 ) } <f>(p N _ 1 )} = 

<f>(p N -i)i we have 

r <uj(p N _ 1 ,p N ) (41) 

where 



rB < { 



^{Pn-^Pn 



{ HPn-i ) + KPn-i ,P n ) if Pn-i +Pn <8 

max{V>fe + P N . 1 ),<l>(p N . 1 )} + KPn-hPn) ^Pn-i + Pn > 8,P N _ t < 8 

maxjV'fe +P n _ 1 )^Kp n _ 1 )} + KPtr.^Ptr) if P N ^ > 8,p N _, + P N < 1/3 

, max{V'(p JV _ 1 ),^(l/3)} + l(p N _ 1 ,p N ) ifp N -i + Pn > V 3 



The above results can be summarized in the following theorem 

Theorem 5 Let S = (p\ } . . . } p N _ 17 p N ) be a discrete source. The redundancy of the 
corresponding Huffman code is upper bounded by: 

r<l-H(p N ) ifp N _ 1 +p N = l (42) 

r < l+p N _ 1 +p N -H(l-p N _ 1 -p N ,p N _ 1 ,p N ) if(l-p N )/4: <p N _ t < (l-p N )/3 (43) 

r < max{l+p JV _ 1 + p N - H(l - p N _, - p N ,p N _ 1 ,p N ),to{p N _ 1 ,p N )} if p N _, < (1 -flJ/4 

(44) 

Notice that if p N _ ± + p N < 1 then p N _ ± < (1 - p N )/3. 
Figure 5 shows bound (44). 

Following a similar reasoning but using 1 — 7i-(p N ) for upper bounding r# instead 
of (3) we get the bound provided by Yeung. Therefore it is clear that the bound (41) 
is sharper than the bound of Yeung, since bound (3) is sharper than 1 — 7i(p N ). 
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Figure 5: An Upper bounds on the redundancy r as functions of p N _ x and p N 
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